What Can Be Known about the Radiometric Response from Images?

Brightness values of pixels in an image are related to image irradiance by a non-linear function, called the radiometric response function. Recovery of this function is important since many algorithms in computer vision and image processing use image irradiance. Several investigators have described methods for recovery of the radiometric response, without using charts, from multiple exposures of the same scene. All these recovery methods are based solely on the correspondence of gray-levels in one exposure to gray-levels in another exposure. This correspondence can be described by a function we call the brightness transfer function. We show that brightness transfer functions, and thus images themselves, do not uniquely determine the radiometric response function, nor the ratios of exposure between the images. We completely determine the ambiguity associated with the recovery of the response function and the exposure ratios. We show that all previous methods break these ambiguities only by making assumptions on the form of the response function. While iterative schemes which may not converge were used previously to find the exposure ratio, we show when it can be recovered directly from the brightness transfer function. We present a novel method to recover the brightness transfer function between images from only their brightness histograms. This allows us to determine the brightness transfer function between images of different scenes whenever the change in the distribution of scene radiances is small enough. We show an example of recovery of the response function from an image sequence with scene motion by constraining the form of the response function to break the ambiguities. 1 Radiometric Calibration without Charts An imaging system usually records the world via a brightness image. When we we interpret the world, for example if we try to estimate shape from shading, we require the scene radiance at each point in the image, not just the brightness value. Some devices produce a brightness which is a linear function of scene radiance, or at least image irradiance. For most devices, such as consumer digital, video, and film cameras, a non-linear radiometric response function gives brightness in terms of image irradiance.1 1 We are ignoring spatially varying linear factors, for example, due to the finite aperture. We will assume that the response function is normalized both in domain (irradiance) and range (brightness). A. Heyden et al. (Eds.): ECCV 2002, LNCS 2353, pp. 189–205, 2002. c © Springer-Verlag Berlin Heidelberg 2002 190 M.D. Grossberg and S.K. Nayar Some vision applications such as tracking may not require precise linear irradiance values. Nevertheless, when one estimates the illumination space of an object as in [1], estimates the BRDF from images [2], determines the orientation of surface normals [3], or merges brightness images taken at different exposures to create high dynamic range images [4], one must find irradiance values from brightness values by determining the radiometric response of the imaging system. We can recover the radiometric response function by taking an image of a uniformly illuminated chart with patches of known reflectance, such as the Macbeth chart. Unfortunately, using charts for calibration has drawbacks. We may not have access to the imaging system so we may not be able to place a chart in the scene. Changes in temperature change the response function making frequent recalibration necessary for accurate recovery of the response. Additionally, we must uniformly illuminate the chart which may be difficult outside of a laboratory environment. The problems associated with using charts have lead researchers to develop methods to recover the radiometric response from registered images of a scene taken with different exposures. Mann and Picard assume the response function has the form of a gamma curve [5]. They estimate its parameters assuming they know the exposure ratios between the images. Debevec and Malik also assume the ratio of exposures is known, but they recover the log of the inverse radiometric response without a parametric form [6]. To obtain a solution they impose a smoothness constraint on the response. Mitsunaga and Nayar assume that the inverse response function can be closely approximated by a polynomial [4]. They are then able to recover the coefficients of that polynomial and the exposure ratios, from rough estimates of those ratios. Tsin, Ramesh and Kanade [7] and separately, Mann [8] recover both the response and exposure ratios by combining the iterative approach from [4], with the non-parametric recovery in [6]. The essential information all methods use for recovery is how brightness graylevels in one image correspond to brightness gray-levels in another. Ideally a function we call the brightness transfer function describes this correspondence.2 Figure 1 illustrates the role of the brightness transfer function. All chart-less recovery methods [4, 5, 6, 7, 8] use the constraint that all irradiances change by the exposure ratios to recover response function and exposure ratios from exactly the information contained in the brightness transfer function. Previous authors have not completely addressed whether there actually is enough information in the images to recover the response function and exposure ratios from images without charts. Put another way, since the brightness transfer function contains all the information about the response function, we could ask: given the exposure ratio, is there a unique response function for each brightness transfer function? We show that solutions to this inverse problem are not unique. We demonstrate that due to a fractal-like ambiguity any method to recover the response function must constrain the function, for example with regularity, 2 Most methods use all pairs of gray-levels at corresponding points in the images. Mann [8] shows that the brightness transfer function summarizes this information. What Can Be Known about the Radiometric Response from Images? 191 Fig. 1. A diagram showing the role of the brightness transfer function. All the information in the images, relevant for chart-less recovery is contained in the brightness transfer functions. These describe how a brightness in one image corresponds to a brightness in another image. Thus, the problem of recovering the radiometric response function falls into two parts: recovery of the brightness transfer functions from images, and recovery of the radiometric response function and the exposure ratios from the brightness transfer functions. to break this ambiguity. Beyond this ambiguity, is it possible to recover the response function and the exposure ratios simultaneously and uniquely? We show that there are families of solutions, arising from what we call an exponential ambiguity. Again, only by making assumptions on the response function can we expect a unique solution. Can we recover the exposure without recovering the response? We will show when this is possible. Given that it is possible to recover the response function and exposure ratios from the brightness transfer function by making assumptions on the form of the response, how do we recover the brightness transfer function from images? Previous work compared registered images of a static scene taken at different exposures to recover the brightness transfer functions. Is it necessary for the scene to be static and do we require any spatial information to recover the brightness transfer function? It is not, since we show that the brightness transfer function can be obtained from the histograms of the images. This implies that in situations where the distribution of scene radiances remains almost constant between images we can still recover the brightness transfer function. To illustrate this, we recover the response function from a sequence of images with scene motion. 2 The Fundamental Constraint for Chart-Less Recovery The brightness value at a point in the image should allow us to determine the scene radiance. The ideal brightness value I is linear in the scene radiance L. The ideal brightness is related to scene radiance by I = LPe, where P is a factor due to the optics of the system, and e is the exposure, following the notation of [4]. For a simple system, P = cos4 α/c2, where α is the angle subtended by 192 M.D. Grossberg and S.K. Nayar the principle ray from the optical axis and c is the focal length.3 The exposure is given by e = (πd2)t, where d is the size of the aperture and t is the time for which the photo-detector is exposed to the light. Even though e contains the integration time, t, we can think of the ideal brightness I as image plane irradiance. A function f called the radiometric response function relates the actual measured brightness value M = f(I) at a photosensitive element to the image plane irradiance I. Imaging system designers often intentionally create a non-linear radiometric response function f , to compress the dynamic range, for example. Since measured brightness indicates relative irradiance, we can assume that the response function f monotonically increases.4 The minimum irradiance is 0, while the maximum irradiance, Imax is a single, unrecoverable parameter. Thus we normalize domain of f, irradiance I, to go from 0 to 1. We also normalize the range of f , brightness M , so that f(1) = 1 and f(0) = 0.5 Up to this normalization, we can determine f if we take an image of a uniformly illuminated chart with known reflectance patches. Without a chart we must find constraints that permit us to extract f from images without assuming the knowledge of scene reflectances. As a special case of what we mean, suppose we take images A and B of the same scene with different exposures eA and eB . If image A has image irradiance IA at a point and the corresponding point in image B has the irradiance IB , then IA/eA = IB/eB . The exposure ratio k := eB/eA expresses the relationship between the two images, IB = kIA. Relating this back to measurements in images A and B we have f(IA) = MA and f(IB) = MB . Monotonicity of radiometric response function makes it invertible. Let g := f−1 be the inverse of f . Then, we have the equation, g(MB) = kg(MA). (1) All chart-less methods base recovery of g and k on this equation. In each pair of images, each corresponding pair of pixel brightness values gives one constraint. The exposure ratio k is constant for each pair of images. When k is known, this gives a linear set of equations g. If g is a polynomial, then equation (1) becomes linear in the coefficients of the polynomial. Mitsunaga and Nayar solve for these coefficients [4]. Debevec and Malik [6] and Mann [8] take the log of both sides of equation (1). Rather than start with a parameterized model of log g they discretely sample it at the gray-level values, treating it as a vector. By imposing a regularity condition on the discrete second derivatives of log g, they are able to obtain a solution. 3 Details of P for a simple perspective camera can be found in Horn [9]. Whereas Mitsunaga and Nayar [4] discuss selection of pixels in the image where P is nearly constant, we will assume P is constant throughout the part of the image we analyze. 4 A monotonically decreasing response function is also possible, as in negative films. We can, however, re-normalize f so that f increases with irradiance, for example by multiplication by −1. 5 The minimum brightness value in digital imaging systems is often effectively greater than zero due to non-zero mean thermal noise called dark current. By taking an image with the lens covered, this effect may be estimated and subtracted. What Can Be Known about the Radiometric Response from Images? 193 When we know g but not k, we can solve equation (1) for k. Mitsunaga and Nayar [4] and Mann [8] use an iterative scheme in which they first solve for g with an initial guess for k. Updating their estimates, they iteratively solve for k and g. 3 Brightness Transfer Functions The pairs of measurements MA and MB at corresponding points in different images of the same scene constitute all the information available from which to recover the response function in chart-less recovery. Mann [8] pointed out that all this information is contained in a two variable histogram he calls the comparagram. If (MA,MB) are any two pairs of brightness values, then J(MA,MB) is the number of pixels which have brightness value MA in image A and MB at the corresponding point in image B. The comparagram encodes how a gray-level in image A corresponds to graylevel in image B. For real images, a probability distribution most accurately models this correspondence, rather than a function. A function fails to model all the pixel pairs because of noise, quantization of the brightness values, spatial quantization, and saturated pixels. Ignoring these considerations for a moment, from equation (1), we see that a function should ideally relate the brightness values in the images MB = T (MA) := g−1(kg(MA)) (2) which we call the brightness transfer function. This function describes how to transfer brightness values in one image into the second image. We can estimate the brightness transfer function T from J .6 For a collection of images A1, A2, A3, . . . Al the brightness transfer functions of all possible pairs Tm,n summarize the correspondence of gray-levels between the images. Once we estimate T , we have a modified version of equation (1), given by g(T (M)) = kg(M). (3) This equation has an advantage over equation (1): because the function T contains all information about the gray-level correspondence between images, we can study the mathematical problem of existence and uniqueness of solutions to equation (3). To study solutions to equation (3) we must first derive some properties of brightness transfer functions. From equation 2 we know that the monotonicity of g implies the monotonicity of T and so T−1 exists. Define T 0(M) := M , T(M) := T (Tn−1(M)) and T−n(M) := T−1(T 1−n(M)). Theorem 1 (Properties of the Brightness Transfer Function). Let g be a smooth monotonically increasing function with smooth inverse. Suppose that g(0) = 0 and g(1) = 1, and k > 1, then the function T (M) := g−1(kg(M)) 6 Mann [8] calls T the comparametric function, and the process of going from J to T , comparametric regression. 194 M.D. Grossberg and S.K. Nayar has the following properties: (a) T monotonically increases, (b) T (0) = 0, (c) limn→∞ T−n(M) = 0, and (d) if k > 1, then M ≤ T (M). [see appendix A for proof]. Assuming that k > 1 just means that we order our images so that exposure increases. For example, for a pair of images with 0 < k < 1, we can replace equation (3) with the equation f(T−1(M)) = (1/k)f(M), where (1/k) > 1. By replacing k with 1/k and T with T−1, we have reordered our images so that k > 1. To order the images themselves note that when k > 1, then g−1(kI) ≥ g−1(I) since g−1 monotonically increases. In other words every brightness value in one image corresponds to a brighter value in the other image. Therefore, for k > 1, T goes from the image with darker average pixel value, to the image with lighter average pixel value.7